perf(ci): batch pulp delivery and promotion into single modify tasks#3557
perf(ci): batch pulp delivery and promotion into single modify tasks#3557kduret wants to merge 28 commits into
Conversation
Waiting for each upload task before sending the next upload pays the pulp task-queue latency once per package; under concurrent deliveries the queue grows and a single slow task aborts the delivery (~10 min wait_task cap) after minutes of sequential waits. Collect the task hrefs during the upload loop and await them as a batch before publishing (new wait_tasks helper); the batch guard aborts only when no task completes for ~10 min, so a draining queue is not mistaken for a hang. Refs MON-200796
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The GitHub Actions OIDC token expires ~5 minutes after issuance, so any delivery longer than that started failing mid-loop with HTTP 401 (full plugins delivery: ~5400 uploads across 8 jobs died at exactly +5 min), and task polls with a stale token read as 'did not complete in time'. Refresh the token before it goes stale from the api helpers (upload attempts and task-wait sweeps), propagate it to the following steps and rewrite the pulp-cli config so the pulp commands keep working; setup-pulp-cli exports the issuance time and audience the refresh needs. Refs MON-200796
pulp_upload runs in a command substitution, so its internal token refresh only lives in the subshell: the parent shell kept the stale token, killing the rpm stable-guardrail curl with a 401 at +5 min and forcing every post-expiry deb upload to re-mint a token and rewrite the pulp-cli config (seconds per package). Refresh once per loop iteration from the parent shell instead — the guardrail and the inherited subshell token stay fresh, and the per-package refresh overhead disappears. Refs MON-200796
Large deliveries can wait minutes to hours on the server task queue with no output. Emit a progress heartbeat from wait_tasks, at most every 30 seconds (time-based: a single sweep over a large batch can take minutes on its own), counting the completed tasks over the batch total. Refs MON-200796
… E2BIG) Two failures surfaced by a 675-package delivery once the task queue drains: - refresh_pulp_token echoed the ::add-mask:: workflow command on stdout; when the refresh triggers inside pulp_upload's command substitution the command line is captured into the task href, which then never resolves (the batch wait stalls on a single corrupted href). Emit it on stderr. - manifest_write passed the packages array as an --argjson argument, which exceeds the kernel argv size limit with hundreds of packages (jq: Argument list too long, exit 126, right after a successful publish). Slurp the entries from stdin instead. Refs MON-200796
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The published-metadata resolvers piped the index body into an awk that exits on the first match, so the printf writer took a SIGPIPE and bash printed one "printf: write error: Broken pipe" diagnostic per verified package. Read the index from a temporary file instead - same early exit, no pipe writer. Refs: MON-200796
A single limit=1000 page silently truncates once the repository holds more module packages than the page size (the shared apt-plugins repository is past 5000), flagging delivered packages as missing. Refs: MON-200796
List the module packages newest first and stop as soon as every expected package has been seen: the shared plugins repository holds 10k+ module packages, so an unbounded offset walk is slow, eventually fails server-side, and the expected packages sit in the tail with the default ordering. The unexpected-extras report becomes count-based for the same reason. Refs: MON-200796
grep -q exits on the first match, the printf writer takes a SIGPIPE, and pipefail turns the pipeline status into 141: every package that IS present gets flagged missing once the listing outgrows the pipe buffer (the shared plugins repositories are past 12k module packages). Write the listing to a file and let grep read the file - no writer left to kill. Refs: MON-200796
…ng wait) pulp-cli reads its token once at startup, so its built-in wait dies with "Authentication failed for tasks_read" as soon as the publication of a large repository outlives the OIDC token validity (~5 minutes). Start the publication in the background and poll it with wait_task, which re-authenticates on every attempt. Refs: MON-200796
The retry window only covers publication propagation: once a round reads the published metadata and resolves nothing new while some packages already resolved, the remaining ones are not part of the publication at all (e.g. evicted by the retention policy) - report right away instead of burning the whole 5-minute window, and log the per-round progress. Refs: MON-200796
One flaky HEAD out of hundreds (content-app/S3 hiccup) must not fail the whole verification. Refs: MON-200796
Replace the one-task-per-package association (serialized on the pulp repository lock) with the batched flow validated at full scale on the plugins matrix (8 distributions, 675 packages each, delivery + promotion, zero error): - delivery: packages are uploaded as repository-less content (create tasks parallelize across pulp workers), resolved concurrently, then associated with a single repository modify task followed by one publication; - deb flavor: the first package of each architecture goes through the legacy repository code path to materialize the suite associations, the rest of the batch gets its PackageReleaseComponent units created synchronously before the single modify; - promotion: the testing packages are listed (latest build only), their stable suite associations deduced, and promoted with a single modify task instead of one download/re-upload task per package. Refs: MON-200796
The promotion sweep selected testing packages by their relative_path prefix (pool/testing/<module>/). Content reused across suites breaks that assumption: a byte-identical package first delivered to another suite is deduplicated by pulp and keeps its original upload path, so the sweep missed it (20 of 27 jammy perl-cpan-libraries packages in the pulp-only test PR). Membership now comes from the testing suite's PackageReleaseComponent associations. The release-component hrefs themselves are now resolved directly from the release_components listing (?distribution=<suite>&component=main), opened to authenticated users by the pulp bootstrap (delivery-tooling#230), instead of being deduced from the first package's associations - a deduction that was likewise ambiguous for reused content. Refs: MON-200796
The release_components viewset ships without an access policy upstream (pulp_deb 3.9, admin-only fallback) and pulpcore offers no way to create one, so the previous commit's direct listing can never work for the CI user. Replace it with mechanisms built on endpoints the CI user can read: - promotion candidates are selected by joining the module content listing with the published dists/<testing suite> Packages indexes on sha256 (fail-closed on index fetch errors), instead of trusting relative_path; - when every candidate is already published in the stable suite, the promotion short-circuits to a republish (idempotent reruns); - the delivery picks a FRESH package (content not yet in pulp) as the legacy representative of each architecture whenever possible: its single post-upload association identifies the suite release component exactly; reused representatives fall back to a before/after association diff; - the promotion deduces the stable release component from the same before/after diff on its reference package, subtracting the associations of not-yet-promoted candidates on reruns. Refs: MON-200796
The guarded *business-internal distributions answer 403 to anonymous requests, so the delivery verification could not check the served metadata or fetchability of business modules (map showed 3/3 present, 0/3 resolvable), and the deb suite-index reads would fail on guarded suites. Every content-endpoint fetch now goes through content_curl, which sends the same GitHub OIDC credentials as the api calls (the CI user holds the guard downloader role, delivery-tooling#232); unguarded distributions ignore the header. Refs: MON-200796
Same change as the feature branch (see its history for the rationale). Refs: MON-200796
Refs: MON-200796
Align the shared pulp delivery actions with centreon-plugins: the deb plugins stable packages live in a dedicated repository (apt-plugins-stable / ubuntu-plugins-stable) with plain-codename suites, matching the artifactory client layout. For every other deb family the new stable_repository_name/stable_base_path outputs default to the delivery repository, so behavior is unchanged. Also picks up the latest shared-script fixes (index fetch retries, partial-rerun release component fallback). Refs: MON-200796
…etry Align with centreon-plugins: convergent reruns of an interrupted deb promotion (stable membership from the repository version for dedicated stable repositories, release component recovery, republish-and-recheck last resort), segment-aware latest-build selection replacing the plain string max, and a shared retried start_modify_task helper for every repository modify call. Refs: MON-200796
Align with centreon-plugins: under a full-matrix delivery the publication of a large repository can queue for minutes before its execution; the ~10 min cap aborted jobs whose task then succeeded. Refs: MON-200796
Align with centreon-plugins: duplicate suite-association creates on a rerun are logged as an explicit INFO (the lookup fallback resolves them); a WARN with the response body remains for real failures. Refs: MON-200796
Align with centreon-plugins: the api has been observed answering stale empty pages for content committed seconds earlier; lookup fallbacks now retry empty results with backoff. Refs: MON-200796
Align with centreon-plugins: a gateway error page behind a 2xx made the jq parse silently kill the association subshell under set -e; the parses are failure-tolerant and fall through to the lookup path. Refs: MON-200796
Align with centreon-plugins: legacy uploads and repository modify retry when their task fails on the shared-repository version bookkeeping race (duplicate version number or retention-cleanup foreign key conflict). Refs: MON-200796
Align with centreon-plugins: the early exit requires 6 consecutive stalled rounds and the resolution window covers the content-app cache TTL, so index lag no longer reads as missing packages. Refs: MON-200796
Align with centreon-plugins: 'Worker has gone missing' task failures are retried like the repository-version races, including a publication re-create. Refs: MON-200796
Description
Speed up and harden the Pulp delivery and promotion by replacing the one-task-per-package flow with batched single-
modifyoperations.Pulp serializes the tasks of a repository server-side (per-repository lock). With one association task per package, a full delivery paid the queue latency N times: on a develop centreon-plugins run (8 distributions in parallel), deliveries timed out, and the sequential promotion was estimated at ~90 minutes per distribution.
Changes
Delivery (
deliver-rpm.sh/deliver-deb.sh):modifytask, followed by one publication.Promotion (
promote-rpm.sh/promote-deb.sh):modifytask instead of one download/re-upload task per package (~2-3 min instead of ~90 min per distribution).Shared (
api.sh,check-*.sh):Validation
Validated at full scale on the plugins matrix: 8 distributions in parallel with no concurrency cap, 675 packages each, delivery to testing + promotion to stable,
675/675 present · 675/675 resolvableon every repository, zero error and zero warning (run 29748876765).Applied identically to centreon, centreon-collect, centreon-modules and centreon-plugins (shared pulp files kept byte-identical).
Delivery and promotion flow
The key idea of the batched method: packages are uploaded as repository-less content (parallel, no repository lock), their suite/repo associations are prepared, and everything lands in the repository through one
modifytask creating a single repository version, followed by one publication. Every step is retried and reruns converge (already-uploaded content and already-existing associations are looked up instead of recreated).RPM delivery
flowchart TD A["Built .rpm packages"] --> B{"Version already published in stable?"} B -- yes --> Z["Fail loud: bump the version"] B -- no --> C["Repository-less uploads<br>parallel pool of 8, retried"] C --> D["Resolve the created content hrefs<br>task results, lookup fallback on rerun"] D --> E["ONE repository modify<br>single new repository version"] E --> F["ONE publication"] F --> G["Manifest consumed by the verification step"]DEB delivery
The deb flavor needs the suite structure (
Release,ReleaseComponent,ReleaseArchitecture) and onePackageReleaseComponent(PRC) per package. Therelease_componentsAPI is not listable by the CI user, so the release component href is deduced from the associations of a representative package.flowchart TD A["Built .deb packages"] --> B{"Version already published in the stable suite?"} B -- yes --> Z["Fail loud: bump the version"] B -- no --> C["Legacy upload of the FIRST package per architecture<br>repository code path: creates the suite structure"] C --> D["Deduce the suite release component<br>diff of the representative's PRC set<br>before/after the upload"] D --> E["Repository-less uploads of the remaining packages<br>parallel pool of 8, retried"] E --> F["Create one PRC per package<br>synchronous parallel creates, lookup fallback on rerun"] F --> G["ONE repository modify<br>packages + PRCs in a single repository version"] G --> H["ONE structured publication"] H --> I["Manifest consumed by the verification step"]Promotion to stable (deb)
Two layouts: standard/business promote into a stable suite of the same repository; plugins promote cross-repository into a dedicated
apt-plugins-stable/ubuntu-plugins-stablerepository whose suite is the bare codename (bookworm main), keeping the client apt configuration identical to artifactory.flowchart TD A["Module packages in the repository version<br>selected by the module label"] --> B["Keep packages published in the TESTING index<br>latest build per package and arch<br>segment-aware version comparison"] B --> C{"All candidates already promoted?"} C -- yes --> R["Republish only and stop"] C -- no --> D["Legacy upload of the first candidate per arch<br>into the STABLE suite of the stable repository"] D --> E["Deduce the stable release component<br>PRC diff, dedicated-repo lookup,<br>republish-and-recheck as last resorts"] E --> F["Create the missing stable PRCs in parallel"] F --> G["ONE modify into the stable repository"] G --> H["ONE structured publication + manifest"]The rpm promotion is the simpler variant of the same shape: label lookup, latest-build selection, one
modifyinto the stable repository, one publication.Refs: MON-200796